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Question From the Field 


How are states supporting principal preparation programs in measuring and 
using program results to build a strong principal pipeline? 

Parallel efforts in principal evaluation and educator 
preparation are prompting new conversations among 
states, higher education programs, private foundations, 
and professional associations about strengthening school 
leadership development pipelines. A strong school principal 
pipeline has many elements, but each element supports 
the evaluation and development of current and future 
school principals; these elements include common, 
aligned leadership standards; performance measures; 
and professional learning supports. Taken together, a 
strong principal pipeline guides and grounds principal 
recruitment, preparation, hiring, evaluation, and support 
(Turnbull, Riley, & MacFarlane, 2013). School districts and 
principal preparation programs both have separate but 
related roles in improving principal pipelines; however, 
because separate organizations are responsible for 
supporting principals’ careers, P-20 partnerships or state 
agencies coordinate principal pipeline improvement efforts. 

In this Ask the Team brief, we highlight state activities and 
pertinent research on principal preparation, specifically 
state efforts to measure principal preparation program 
effectiveness. State efforts to create models for leadership 
program accountability face numerous methodological 


^ A 

What is a principal preparation program? 

In this brief, we define a principal 
preparation program as a state-accredited 
program of study that fully or partially 
prepares educators for certification as 
a school principal. 

What is program approval 
and accreditation? 

In this brief, we define program approval 
and accreditation as the state’s regulatory 
framework defining standards, reporting 
requirements, and review processes that 
must be met by: 

■ New preparation programs before 
initiating candidate recruitment 
(initial approval). 

■ Established preparation programs 
to continue qualifying graduates 
for either partial or full professional 
accreditation (reapproval). 
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challenges. To inform collaborative efforts across a variety of policymakers and stakeholders, we 
reviewed information on principal preparation accountability models, examined existing reports on 
pilot approaches in states, and identified existing partnerships that are tackling this challenging 
work. In the sections that follow, we offer information on the following: 

■ State of the state: What measures do most states say they collect and how do they use 
the data? 

■ Testing the waters: What new accountability models are states piloting? 

■ Charting your course: What are some strategies for establishing a collaborative model? 

We offer these examples as a starting point for discussions on principal preparation program 
effectiveness, continuous improvement, and accountability. The Center on Great Teachers and 
Leaders (GTL Center) does not endorse any of the 
models described in the brief but rather offers them 
for informational purposes only. 

Introduction. Our search indicates that principal 
preparation programs ascribe to common standards for 
principal professional practice, but the programs do not 
commonly have performance measures that are tied to 
principals’ postgraduation experiences. State education 
agencies, districts, and national organizations are 
currently collaborating to develop valid and reliable 
measures of how preparation program content, clinical 
experiences, and training supports provide optimal 
learning opportunities to future principals. 

Our review identified few principal preparation policy 
reviews or accountability models. Previous reviews of 
state accountability and approval systems for leadership 
preparation have identified an overall lack of rigor in 
program accountability measures and inadequate data 
on program impacts (Briggs, Cheney, Davis, & Moll, 

2013). No widely accepted model of preparation program 
accountability exists and few research studies find 
connections between principal preparation program 
features and principal success, school performance, or 
improved school culture. This lack of guidance makes 
collaboration among multiple actors — state education 
agencies, policymakers, institutions of higher education, 
national associations, districts, and foundations — crucial 
in identifying new models for leadership preparation 
accountability and program evaluation. 


™ \ 

Report to Note 

The Reform Support Network’s (RSN’s) 
Promising Practices in Approving 
and Renewing Principal Preparation 
Programs (February 2014) offers five 
recommendations for state leaders 
interested in improving principal preparation. 
Based on policy analysis and interviews 
with several states' leaders, RSN compiled 
a set of helpful considerations: 

■ “Create clear and comprehensive 
expectations for programs. 

■ Establish specific, quality-focused 
fieldwork criteria. 

■ Plan how to collect and use 
outcome data. 

■ Conduct site visits for approving 
and renewing programs. 

■ Clearly indicate requirements on the 
application for new programs.” (p. 1) 

The report includes links to example 
approval frameworks and guidelines 
from several states. 

Source: http://www2.ed.gov/about/inits/ 
ed/implementation-support-unit/ 
ech-assist/promising-practices-principal- 
preparation.pdf 
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r |J STATE OF THE STATE 

What measures do most states say they collect and how do they 
use the data? 


States vary widely in how they assess principal preparation program quality and effectiveness, but 
they typically do so as part of program approval and renewal processes (Briggs et al. , 2013). To help 
clarify how different categories of measures are related to each other, we have grouped them into 
two types that we will use throughout the brief when describing state approaches to measurement: 

■ Program Characteristics: descriptions of program features and rationales that comprise 
the principal candidate’s learning experience, such as candidate selection criteria, program 
standards, program documents, candidate completion requirements, curriculum and 
coursework, or clinical experience descriptions. 

■ Program Outcomes: measures of immediate program impact on the principal pipeline, such 
as rates of graduate licensure, candidate graduation and certification rates, job placement — 
especially in high-need schools — and principal retention. Other outcome data could include 
candidates’ persistence in the program, candidates’ success in the program (such as GPA), 
and candidate graduation rates. 

Program outcomes also can include the principal preparation graduates’ effectiveness as 
school leaders after leaving the program, such as performance evaluation ratings, which 
might provide evidence of the following: impact on teacher effectiveness and retention, 
school climate, and student achievement and learning, or information from other data 
sources, including employer satisfaction surveys, graduation rates, and student retention. 



Center for the Evaluation of Educational Leadership Preparation and Practice (CELP) at the University Council for 
Educational Administration (UCEA) 

Principal preparation programs vary in both their intended program outcomes and their approach to gathering 
information about program effects. The CELP initiative is intended “to make available valid and reliable evaluation 
research tools, methods, and training materials and strategies for leadership preparation programs" that have a 
research base and are useful to the continuous improvement of programs. In collaboration with UCEA members 
throughout the United States, the center offers several program evaluation tools, including formative and summative 
evaluation planners and INSPIRE surveys that provide 360-degree data on program quality and graduate impacts on 
leadership, school conditions, and student learning. 


Source: http://edleaderprep.org/ 
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Partnership 
. Spotlight 


Alliance to Reform Education Leadership (AREL) at the George W. Bush Institute 

The Alliance to Reform Education Leadership (AREL) at the George W. Bush Institute is a network of 28 principal 
preparation programs in 15 states and the District of Columbia working on measuring program effectiveness. AREL’s 
network includes 28 principal preparation programs. The preparation programs were selected and assessed against 
AREL’s Nine Principal Preparation Competencies, which include a requirement for data collection as part of program 
evaluation: Each of the 28 programs is sorted into one of three categories based on their level of maturity, 
effectiveness, and data collection: 

■ Exemplary programs have demonstrated success over time and are considered among the best in the country. 

The programs are currently collecting and evaluating outcome data on the impact of their graduates on student 
achievement and seeing positive results. 

■ Accelerating programs have adopted the AREL leadership program element standards and are beginning to collect 
data on program graduates. 

■ Start-up programs have adopted the AREL leadership program element standards but are still in the process of 
recruiting and graduating candidates. 

The AREL network has begun its work by assessing state-level capacity for collecting and using student learning, school 
demographic, and human resources information for principal preparation program accountability, and is working on an 
evaluation of its network programs. 

Source: http://www.bushcenter.org/alliance-reform-education-ieadership/arel-network 


In 2012, AREL launched the Principal Policy State Survey, which asked all 50 chief state school 
officers about a range of principal policies in their states, including principal workforce data collection 
and state requirements for principal preparation program approval (Briggs et al. , 2013). AREL’s first 
report summarizing the survey results, Operating in the Dark: What Outdated State Policies and Data 
Gaps Mean For Effective School Leadership, found that: 

■ States generally do not use their existing oversight, licensure, and data-monitoring powers 
to improve school leadership pipelines. 

■ Many states lack the most basic data for monitoring the supply of principals for schools, 
such as the number of candidates graduating from programs and obtaining licensure. 

■ Few states link data on principal effectiveness in schools back to the education programs 
that prepared them as school leaders. 

■ Most states are currently unable to distinguish their most successful preparation programs 
from their least effective ones, which hampers accountability efforts and strategic efforts to 
improve principal quality. 

Relying on AREL’s survey data, in Figure 1, we provide a brief overview of the number of states 
that reported using several different measures of program characteristics as part of their program 
approval process. 1 


1 The full survey findings are presented in AREL's Operating in the Dark: What Outdated State Policies and Data Gaps Mean 
for Effective School Leadership. 
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Figure 1. Types of Information on Program Characteristics Required for Program Approval 
(State Self-Report) 2 
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The chief state school officers responding to the survey noted that the state gathered program 
quality information on principal preparation programs. As Figure 1 indicates, most states report 
requiring information on preparation program standards or competency frameworks, program 
purpose, and program completion requirements. More than half of states also examine program 
coursework and context; information about candidate selection criteria and processes is less 
commonly collected — less than half of states say they do so. Information on recruitment efforts, 
program evaluation practices, clinical leadership experiences, and postgraduate supports is 
collected in less than a quarter of all states. 

With the exception of licensure attainment rates for 
programs, few chief state school officers commented 
that their state gathered program outcomes data. 

Table 1 describes the types of outcome measures that 
states reported collecting and using as part of principal 
preparation program approval and reapproval. Between 
one third and one quarter of all states report collecting 
data for outcome measures, and even fewer use the data 
for program approval or reapproval. 


Looking for State- by-State Data? 

AREL offers two ways to view the survey 
results for individual states: You can 
download individual, in-depth state 
profiles for each state from AREL's 
interactive, online map or you can 
download the information for all 
50 states in a single table. 

V 


2 Note: The AREL survey questions on program approval are structured around AREL's “Nine Principal Preparation 
Competencies”; because the data are based on state self-report, states likely use additional program characteristics 
measures beyond those represented in Figure 1. 
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Table 1. Types of Outcome Measures and Uses (State Self-Report) 



Ensure Data are 
Collected* 

Number of States 

Use the Measure 
for Initial Program 
Approval 

Use the Measure for 
Program Reapproval 

Program Outcome Measures 

1. Principal job placement rates 

19 

7 

9 

2. Principal retention rates in the job 

13 

5 

6 

3. Graduates earning licensure 

N/A 

24 

N/A 

4. Principal impact on student learning or achievement 

16 

8 

14 

5. Principal evaluation 

16 

13 

11 


^Either the state collects the data or requires preparation programs to collect the data. 



TESTING THE WATERS 

What New Accountability Models are States Piloting? 


As the AREL report highlighted, most states limit existing evaluation and approval processes to 
measures of program characteristics. In partnership with other organizations, states are convening 
stakeholder groups to discuss improving educator preparation accountability. One part of these 
multifaceted conversations has focused on evaluation 
methods for principal preparation programs, including 
possible outcome variables for states to use as part of 
an accountability model. 

An emerging approach is creating annual report cards that 
are compiled by state education agencies in concert with 
preparation programs. In the wake of federal Race to the 
Top awards, several states, such as Florida, Louisiana, 

North Carolina, and Tennessee, committed to developing 
stronger mechanisms for preparation program accountability 
that included linking preparation programs with their 
graduates’ performance evaluation data after placement. 

Although few states are currently far enough along to 
actually produce this data linkage, the proposed report 
cards feature data on a range of program outcomes, 
including program participant satisfaction survey results 
and program graduate performance evaluation data, 
such as impacts on school climate, teacher quality, and 
student learning. For an early example of an online report 
card, check out North Carolina’s Institutions of FHigher 
Education Educator Preparation Program Report Cards. 


Did You Know? 

The Council for the Accreditation of 
Educator Preparation (CAEP) adopted 
new accreditation standards for education 
preparation programs in August 2013. The 
new standards include a stronger 
emphasis on developing a “culture of 
evidence.” To support preparation programs 
in meeting the new standards, the council 
released its CAEP Evidence Guide, which 
explains CAEP’s perspective on what a 
“culture of evidence” looks like in the use 
of data in educator preparation and 
accreditation. The guide includes protocols 
and instructions for collecting and 
analyzing data, as well as guidance on 
what constitutes valid evidence and 
suggestions for collecting evidence on the 
impact of graduates on student learning. 
Source: http://caepnet.org/ 
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Because report cards are compiled annually, they hold promise to provide programs, candidates, and 
state agency staff with timely and actionable information about program effectiveness. Reapproval 
processes in most states occur roughly every five to seven years. The data collected as part of the 
annual report card could contribute to reapproval assessments, but they enable states and programs 
to collaborate in identifying and acting on program successes and challenges as part of a continuous 
improvement process. 


State 

^Spotlight 


card requires collaboration between the Board of Regents, the Ohio Department of Education, and higher education 
institutions to share data in order to link the performance of program graduates after they begin serving in schools. 
The state took a collaborative approach to the development of the report cards, inviting 13 public and 38 private educator 
preparation institutions to participate in the development process.The report card includes the following measures: 

■ Licensure test scores 

■ Value-added data (schoolwide for administrators) 

■ Candidate academic measures 

■ Field and clinical experiences 

■ Preservice candidate survey results 

■ Resident educator survey results 

■ Resident educator persistence data 

■ Excellence and innovation initiatives (which could include placement rates in hard-to-staff schools) 

■ National accreditation 

Source : https://www.ohiohighered.Org/2013_ohio_educator_performance_reports#overview 


Network for Transforming Educator Preparation (NTEP) 

The Council of Chief State School Officers (CCSSO) has selected seven states-Connecticut, Georgia, Idaho, Kentucky, 
Louisiana, Massachusetts, and Washington-to participate in NTER a two-year pilot focused on transforming educator 
preparation and entry systems into the profession. With support from 17 national organizations, NTEP is helping states 
take action in three key policy areas: licensure, program approval, and analyzing and reporting information to improve 
preparation programs (CCSSO, 2013). 

The NTEP pilot builds on policy recommendations developed by CCSSO’sTask Force on Educator Preparation and Entry 
into the Profession, which included nine state chiefs and members from the National Governors Association (NGA) 
and the National Association of State Boards of Education (NASBE).The recommendations include actions that states 
should take to improve the rigor of principal preparation program approval and to better support preparation programs 
in collecting data through multiple measures of program quality and effectiveness. 



Ohio. As part of a broader effort at strengthening program evaluation for educator preparation programs, Ohio’s 
Board of Regents has implemented an annual report card for teacher and principal preparation programs. The report 
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■■ Bonus 

CHARTING YOUR COURSE 

Resource 

Guiding Questions for Establishing a Collaborative Model in Your State 


Developing measures that are sound and useful for preparation program review, assessment, and improvement 
requires a collaborative approach. The key to any successful effort at improving school leader preparation is 
collaboration between state education agency staff, institutions of higher education, preparation programs, and 
districts (Jackson & Kelley, 2002). The following questions are designed for state education agency staff to consider 
as they initiate reforms to principal preparation program accountability. Not all questions may be applicable to your 
state context. Choose the questions that fit your needs. 


Clarify Your Vision and Goals 

□ What legislative or regulatory mandates do you need to ensure that you meet? 

□ What is the primary purpose of your state's efforts in evaluating program effectiveness (e.g., accountability, 
consumer information, continuous improvement)? 

□ What is the broader vision of improving school leadership that will guide your selection of performance measures? 

□ What are the program characteristic outcomes that are high priorities for your stakeholders and your state context? 

Assemble Your Partners 

□ Who should be at the table to ensure you select appropriate measures and that you can eventually collect the 
necessary data to assess program effectiveness? 

□ What commitments should be expected from different stakeholder groups and how can you make those 
clear in advance? 

□ What supports can you provide to stakeholders to promote their participation? 

□ Who are the school leadership preparation innovators in your region or state and how can you invite them to 
share their lessons learned? 


Map Out Policy Alignments 

□ What other principal pipeline policies and initiatives are under way in your state? 

□ What other data systems and education reforms might influence the availability of measures 
(e.g., K-20 longitudinal data, educator evaluation)? 

□ How will you communicate and collaborate across state agencies involved in educator pipeline reform to ensure 
all reforms are part of an aligned and coherent policy? 
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Inventory Your Data 

□ What data does your state already collect that could inform measures of leadership preparation 
program effectiveness? 

□ What is the quality of data already being collected? 

□ What new data might need to be collected? How will you develop or procure the data you need? 

□ How will the data be used (e.g., for accountability, for funding decisions, to provide targeted supports)? 

□ What data do preparation programs already collect and what new data should they collect? 

□ What new data quality assurance processes might need to be developed to improve the accuracy and 
reliability of the data? 

□ What technology or infrastructure barriers need to be addressed to collect data for your selected measures? 

□ Who will collect and house the data? How do your state's regulations on public requests for information impact 
who would be best to collect and house the data? 


Select Your Measures 

□ What sources of evidence will provide accurate and useful information on the most important aspects of 
principal preparation? 

□ How will you ensure the measures selected provide information on key areas of interest for your state? 
(e.g., If high-need schools are a key priority area, are your measures designed to capture data on how many 
principals are placed in high-need schools, whether they stay there, and if they feel they were well prepared 
to lead in that context?) 

□ What information is available about the validity and reliability of the different measures under consideration? 
Are there any potential unintended consequences that might be associated with using the measures? 

□ For measures of principal impact on student learning, what evidence is available that the selected student 
performance measures are reliable but also valid measures of preparation program impact? 

□ Which measures may require a longer timeline for adoption and use (e.g., value-added measurement 
requires a minimum of three years of data on a single program candidate)? 

□ How will the data for each measure be analyzed and combined to make decisions about program quality 
and effectiveness? 
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Plan for Continuous Improvement and Transparency 

□ Who will analyze the data on program effectiveness? 

□ Who is responsible for acting on the data collected on program effectiveness? 

□ Do those responsible for analyzing and using data have the capacity to do so? 

□ What supports are needed to facilitate the process of translating newly collected evidence into improved practice? 

□ How will you monitor the program approval and accreditation system to ensure it is improved on an ongoing 
basis using information on the quality of data and measures being used? 

□ How will you gather rigorous feedback from stakeholders on the program approval and accreditation system 
throughout implementation? 

□ Will you share program performance data with stakeholders? If yes, in what format? 

□ How will you ensure that stakeholders using results from the program approval and accreditation system interpret 
the results appropriately? 

□ How will you assess the impact of the accountability model on preparation program effectiveness? 
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